Hilbert Projection Theorem

Let HH be a Hilbert space ( inner product space that is complete with respect to the norm induced by the inner product) and MM be a finite dimensioal subspace of HH. Then for any x∈Hx \in H, there exists a unique y∈My \in M such that

min⁡m∈M∥x−m∥\min_{m \in M} \|x-m\|

has a unique solution yy. In other words "we can find a unique point in MM that is closest to xx". If m∗m^* is the closest point to xx in MM, then x−m∗⊥Mx-m^* \perp M.

Proof: See lecture notes.

Remark: The proof stated that m∗=x1m^* = x_1 is the closest point to MM. It can also be interpreted as the best approximation of xx choosen from the vectors in MM. The x2x_2 term is the error in the approximation.

Example: Let V=R2V=\mathbb{R}^2 and M=span{[1,1]T}M = \text{span}\{[1,1]^T\}. Find the best approximation of x=[4,7]Tx=[4,7]^T in MM.

Solution: We need to find m∗m^* such that m∗=α[11]m^* = \alpha \begin{bmatrix} 1 \\ 1 \end{bmatrix} and ∥x−m∗∥\|x-m^*\| is minimum.

(x−m∗)⊥M  ⟹  <x−m∗,m>=0∀m∈M(x-m^*) \perp M \implies <x-m^*, m> = 0 \quad \forall m \in M
<x−m∗,[11]>=0<x-m^*, \begin{bmatrix} 1 \\ 1 \end{bmatrix}> = 0
Replace x and m∗ with their values.\text{Replace $x$ and $m^*$ with their values.}
<[4−α7−α],[11]>=0<\begin{bmatrix} 4-\alpha \\ 7-\alpha \end{bmatrix}, \begin{bmatrix} 1 \\ 1 \end{bmatrix}> = 0
Recall that <x,y>=xTy.\text{Recall that $1lt;x,y> = x^Ty$.}
4−α+7−α=04-\alpha + 7-\alpha = 0
α=112\alpha = \frac{11}{2}
m∗=112[11]m^* = \frac{11}{2} \begin{bmatrix} 1 \\ 1 \end{bmatrix}

Example: Let x∈Vx \in V and M=span{v1,v2}M = \text{span}\{v_1, v_2\}. Find the best approximation of xx in MM.

Solution: We need to find m∗m^* such that m∗=α1v1+α2v2m^* = \alpha_1 v_1 + \alpha_2 v_2 that is in the span of MM and ∥x−m∗∥\|x-m^*\| is minimum.

(x−m∗)⊥M  ⟹  (x−m∗)⊥both v1 and v2(x-m^*) \perp M \implies (x-m^*) \perp \text{both } v_1 \text{ and }v_2
<x−α1v1−α2v2,v1>=<x,v1>−α1<v1,v1>−α2<v2,v1>=0<x-\alpha_1 v_1 - \alpha_2 v_2, v_1> = <x, v_1> - \alpha_1 <v_1, v_1> - \alpha_2 <v_2, v_1> = 0
<x−α1v1−α2v2,v2>=<x,v2>−α1<v1,v2>−α2<v2,v2>=0<x-\alpha_1 v_1 - \alpha_2 v_2, v_2> = <x, v_2> - \alpha_1 <v_1, v_2> - \alpha_2 <v_2, v_2> = 0
[<v1,v1><v2,v1><v1,v2><v2,v2>][α1α2]=[<x,v1><x,v2>]\begin{bmatrix} <v_1, v_1> & <v_2, v_1> \\ <v_1, v_2> & <v_2, v_2> \end{bmatrix} \begin{bmatrix} \alpha_1 \\ \alpha_2 \end{bmatrix} = \begin{bmatrix} <x, v_1> \\ <x, v_2> \end{bmatrix}
[α1α2]=[<v1,v1><v2,v1><v1,v2><v2,v2>]−1[<x,v1><x,v2>]\begin{bmatrix} \alpha_1 \\ \alpha_2 \end{bmatrix} = \begin{bmatrix} <v_1, v_1> & <v_2, v_1> \\ <v_1, v_2> & <v_2, v_2> \end{bmatrix}^{-1} \begin{bmatrix} <x, v_1> \\ <x, v_2> \end{bmatrix}
m∗=α1v1+α2v2m^* = \alpha_1 v_1 + \alpha_2 v_2

Example: Let V=R3V = \mathbb{R}^3 and M=span{[1,1,1]T,[1,0,1]T}M = \text{span}\{[1,1,1]^T, [1,0,1]^T\}. Find the best approximation of x=[4,7,2]Tx=[4,7,2]^T in MM.

Solution of Linear Equations

Consider the linear equation as,

Ax=B Ax = B
where A∈Cm×n, B∈Cm×1 and x∈Cn×1 is unknown.\text{where $A \in \mathbb{C}^{m \times n}$, $B \in \mathbb{C}^{m \times 1}$ and $x \in \mathbb{C}^{n \times 1}$ is unknown.}

a) solution exists: when b∈R(A)b \in \text{R}(A)
b) solution is unique: when N(A)={0}N(A) = \{0\}

#EE501 - Linear Systems Theory at METU